Data Pipelines

Back to Home

01. Welcome
02. What is a Data Pipeline?
03. Data Validation
04. DAGs and Data Pipelines
05. Bikeshare DAG
06. Introduction to Apache Airflow
07. Demo 1: Airflow DAGs
08. Workspace Instructions
09. Exercise 1: Airflow DAGs
10. Solution 1: Airflow DAGs
11. How Airflow Works
12. Airflow Runtime Architecture
13. Building a Data Pipeline
14. Demo 2: Run the Schedules
15. Exercise 2: Run the Schedules
16. Solution 2: Run the Schedules
17. Operators and Tasks
18. Demo 3: Task Dependencies
19. Exercise 3: Task Dependencies
20. Solution: Task Dependencies
21. Airflow Hooks
22. Demo 4: Connections and Hooks
23. Exercise 4: Connections and Hooks
24. Solution 4: Connections and Hooks
25. Demo 5: Context and Templating
26. Exercise 5: Context and Templating
27. Solution 5: Context and Templating
28. Quiz: Review of Pipeline Components
29. Demo: Exercise 6: Building the S3 to Redshift DAG
30. Exercise 6: Build the S3 to Redshift DAG
31. Solution 6: Build the S3 to Redshift DAG
32. Conclusion

Back to Home

12. Airflow Runtime Architecture

Airflow components

What are the five components of Airflow’s architecture?

Scheduler

Data Warehouse

Workers

UI/Web Server

Queue

Streaming Server

Database

SOLUTION:

Scheduler
Workers
UI/Web Server
Queue
Database

Airflow UI

What does the Airflow UI do?

Allow the user to construct data pipelines graphically in the UI

Provides a control interface for users and maintainers

Allows the user to write queries against databases

Runs and records the outcome of individual pipeline tasks

SOLUTION:

Provides a control interface for users and maintainers

Scheduler

What does the Airflow Scheduler do?

Runs and records the outcome of individual pipeline tasks

Provides a control interface for users and maintainers

Sends emails on a scheduled basis

Starts DAGs based on triggers or schedules and moves them towards completion

SOLUTION:

Starts DAGs based on triggers or schedules and moves them towards completion

Workers

What do the Airflow Workers do?

Runs and records the outcome of individual pipeline tasks

Provides a control interface for users and maintainers

Sends emails on a scheduled basis

Starts DAGs based on triggers or schedules and moves them towards completion

SOLUTION:

Runs and records the outcome of individual pipeline tasks

Learn Udacity: click here to learn more :)